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(54) Processing image data. 



(57) A plurality of stations (25, 26, 27) are connec- 
ted to a central serving processor (37) via the 
public switched telephone network (33). Each 
station (25, 26, 27) includes a processing device 
(28) and a monitor (29) for generating an in- 
teractive three-dimensional graphics environ- 
ment Similar environments are provided on 
each remote station and interaction between 
the stations is facilitated by transmitting trans- 
formation matrices, modified by one station, to 
all of the other stations. Each transformation 
matrix defines a transformation of an object or a 
light source into the displayable space. 
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1 EPO 

The present invention relates to apparatus and 
methods for processing image data. 

In particular, the present invention relates to ap- 
paratus and methods for displaying three- 
dimensional images. Furthermore, the present in- 
vention relates to an interactive graphics system in 
which a plurality of users may access three- 
dimensional data. 

Systems are known which are capable of synthe- 
sizing two dimensional images in response to data 
defining elements within a three dimensional space. 
The final two-dimensional result may consist of a 
very large number of colored pictured elements (pix- 
els) which may be viewed on a monitor or printed onto 
an image carrying medium. 

In interactive systems arranged to generate data 
representing a three-dimensional space, objects ap- 
pear to move within the three-dimensional space in 
response to input commands. Thus, in such systems 
a machine is required to render a two-dimensional im- 
age from data representing a three-dimensional 
space repeatedly, as the position and/or orientation 
of objects, light sources and view position change in 
response to input commands. Typically, in an interac- 
tive environment, a machine is required to produce 
output images at a rate of between five to twenty per 
second. 

Interactive three-dimensional graphics systems 
have many applications and these applications could 
be extended further if many users could share ac- 
cess to a common three-dimensional world. Thus, in- 
formation could be recorded in three-dimensional 
space and many users, each with their own display 
device, could be given access to this space, allowing 
them to observe the data created by other users and, 
possibly, modify this data. Thus, an interchange of 
ideas could be provided by giving access to shared 
three-dimensional data. 

In interactive graphics systems output images 
are usually displayed on a monitor, similar to a con- 
ventional television receiver. Thus, a processor is re- 
quired to generate a video signal which is supplied to 
the monitor at video rate. Often data is supplied to a 
frame buffer at an interactive rate and image data is 
then read from the frame buffer at video rate, in order 
to generate the video signal. Typically, a look-up table 
is provided, allowing a selection of colors to be de- 
fined in response to selections made by the limited 
number of bits stored in the look-up table. The look- 
up table itself may define particular colors and in- 
clude values representing modifications to lumi- 
nance and saturation. 

Techniques for the transmission of video signals 
are well known and, in addition to one-way television 
broadcasts, it is also known to provide two-way video 
communication, such as in video conferencing facili- 
ties or video telephones. 

However, a problem with transmitting video sig- 
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nals is that a substantial bandwidth is required, there- 
fore transmission costs become high. A further prob- 
lem arises, when sharing access to a three- 
dimensional world, in that additional communication 
5 channels are required for the control of access to that 
world, along with processing facilities to resolve con- 
tention problems. Furthermore, although the video 
image includes a large amount of data, it only repre- 
sents a particular view of the three-dimensional 
10 world, rendered from a particular view point. Thus, 
any attempt to provide multiple access to the three- 
dimensional data is restricted to the particular view 
under consideration. 

According to a first aspect of the present inven- 
ts tion, there is provided apparatus for displaying three- 
dimensional images, comprising a first station having 
a processing device and a display device, and a sec- 
ond station having a processing device and a display 
device, wherein modifications made within a three- 
20 dimensional space displayable by said first station re- 
sult in similar modifications to similar space display- 
able by said second station, each station including 
means for generating a video image from three- 
dimensional data and, on effecting a modification 
25 within the space displayable by said first station, 
means for transmitting modified three-dimensional 
data to said second station over a communication 
channel. 

Thus, the present invention solves the problem of 
30 providing shared access by transmitting modified 
three-dimensional data between stations. 

Three-dimensional data may represent shape 
geometry and transformations for transforming said 
geometry between frames of reference, such as be- 
ds tween a modelling space for each specific object, 
three-dimensional world space and viewing space. In 
a preferred embodiment, data defining the transfor- 
mations of objects are transmitted between stations. 
Possibly, data representing transformation dif- 
40 ferences could be transmitted but in a preferred em- 
bodiment data representing a complete transforma- 
tion is transmitted after said data has been modified. 
Preferably, the transformation data represents ma- 
trices. 

45 In a preferred embodiment a central serving 
processor is provided and a plurality of users are al- 
lowed to modify data under the control of said server. 

The invention will now be described by way of ex- 
ample only, with reference to the accompanying 
so drawings, in which: 

Figure 1 shows an interactive three-dimensional 
graphics station, including a processor and a mo- 
dem; 

Figure 2 shows a plurality of client machine sta- 
55 tions, of the type shown in Figure 1 , connected to 
a server machine; 

Figure 3 is an overview of the operations per- 
formed by a client machine of the type shown in 
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* Figure'2, including communication procedures 
and rendering procedures; 
Figure 4 details the rendering procedures identi- 
fied in Figure 3; 

Figure 5 details the communication procedures 5 

identified in Figure 3; and 

Figure 6 details communication procedures for 

the server machine identified in Figure 2. 

A system for processing image data representing 
three-dimensional objects is shown in Figure 1. A 10 
processor 15 is arranged to write data to and read 
data from an internal memory device. Data stored in 
the memory device defines three-dimensional image 
data, two-dimensional image data and instructions to 
the processor, including instructions relating to com- 15 
munications with a central server. 

The processor 15 receives input data from input 
devices, consisting of a manually operable keyboard 
17 and a position sensitive device, such as a track- 
erball 18, a mouse or a digitising stylus etc. 20 

Two-dimensional images are displayed on a vis- 
ual display unit 19, which receives output image data 
at video rate by raster scanning a frame buffer, often 
provided with proprietary display units. The visual 
display unit 19 has a definition of, typically, one thou- 25 
sand lines with one thousand pixels on each line, re- 
quiring aframe buffer with one million pixel locations. 
For the bulk transfer of program data and image data, 
a mass storage device 20 is provided, such as a hard 
magnetic disk, optical disk or tape drive etc. 30 

The processor 15 is connected to a modem 21, 
which is in turn connected to a conventional public tel- 
ephone line. Thus, modem 21 allows processor 15 to 
communicate over any distance, using the public 
switched telephone network. The processor 15 may 35 
include instructions for initiating a connection to an- 
other system or, alternatively, means for establishing 
such a connection with another modem may be pro- 
vided within modem 21. 

A system embodying the present invention is 40 
shown in Figure 2, in which, in this example, three 
systems of the type shown in Figure 1 are provided. 
A first station 25 includes a first processing device 28 
and a first display device 29. Similarly, a second sta- 
tion 26 includes a second processing device 30 and 45 
a second display device 31, while a third processing 
station includes a third processing device 32 and a 
third display device 33. Each processing device 28, 
30, 32 is connected to the public switched telephone 
network (PSTN) 33 via respective modems 34, 35, so 
36. The PSTN 33 represents the global switched tel- 
ecommunications network, allowing international 
communication. Thus, station 29 could be in Munich, 
station 26 in Washington and station 27 in Tokyo. Al- 
ternatively, other systems may be employed, such as 55 
dedicated lines or higher capacity switched networks, 
such as a packet switching network. 

Each processor 28, 30, 32 of the user stations 25, 
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26, 27 communicates with a central serving proces- 
sor 37 via respective modems 38, 39, 40. Thus, direct 
communication between remote stations is not per- 
mitted and all communications over the network are 
controlled by the serving processor 37. 

The serving processor 37 is shown without an as- 
sociated display device. In some situations a display 
device may be provided, although it is not essential 
to the operation of the system. It is envisaged that ac- 
tual use of the networked environment is made by the 
remote stations. 

However, the server does maintain a complete 
record of all three-dimensional data, so that said data 
may be supplied over the communication network 
when, for example, a new user station is connected 
to said network. 

Three remote stations are shown in Figure 2; it is 
also envisaged that many more stations could be con- 
nected to the network and the extent to which said 
stations may be given access to the graphics envir- 
onment may vary. Thus, some stations may be al- 
lowed to make modifications while other stations 
may only be permitted to view what has already been 
created by others. 

The processes performed by the processor 15, 
for creating an interactive three-dimensional graph- 
ics environment, are shown in Figure 3. These proc- 
esses are performed by processors 28, 30 and 32 as 
shown in Figure 2, identified as client machines, as 
distinct from the serving machine 37 in Figure 2. 

At step 41, communications procedures are 
adopted which allow the processor to communicate 
with the serving processor via the PSTN 33 and their 
respective modems. The operations performed at 
step 41 are detailed in Figure 5. 

At step 42, a question is asked as to whether it is 
necessary to update the display, that is to say, wheth- 
er it is necessary to render the three-dimensional 
data in order to generate new two-dimensional data. 
If this question is answered in the affirmative, step 43 
executes the rendering process, detailed with refer- 
ence to Figure 4. Thereafter, control is returned to 
step 41, allowing further communications to be 
made. Similarly, if the question raised at step 42 is 
answered in the negative, control is returned to step 
41. 

Figure 4 details the rendering operation identi- 
fied as step 43 in Figure 3. At step 45 object culling is 
performed by determining, after local transformation 
and viewing transformation, the extent to which an 
object can actually be seen. Extents are defined for 
the object in each dimension, that is to say, maximum 
and minimum values are determined for the object in 
each of the X, Y and Z dimensions. This process is fa- 
cilitated by defining a cuboid bounding box of planes 
of constant x, y or z extent The bounding box is trans- 
formed into viewing space and the position of the 
transformed bounding box is considered, to deter- 
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mine whetfier.it is totally within the viewable space, 
totally outside the viewable space or intersecting the 
viewable space. 

If an object is totally outside the viewable space 
no further processing is performed on the polygons 5 
defining such an object, thereby making a significant 
saving in processing time. For objects totally within 
the viewing space, processing continues on these 
polygons and no saving is made, however, the proc- 
essing overhead for transforming the bounding box is to 
quite modest. For objects which are partially within 
the viewable space and partially outside the viewable 
space, data defining the planes of intersection are 
stored for reconsideration at step 49, involving the 
clipping of polygons. 15 

A clock is present which effectively gives a 
unique time stamp to each iteration. This stamp is 
then applied each time lighting is moved and each 
time surfaces are moved. Thus, if on a present itera- 
tion, it can be seen that no changes have occured to 20 
the position or orientation of the lighting, or to the 
position of polygons, from the last iteration, lighting 
calculations, calculated on the previous iteration, are 
used again. 

At step 47 the lighting of polygons, where neces- 25 
sary, is performed. In the system shown in Figure 3, 
a local transformation is performed at step 21, sur- 
faces are lit at step 22 and a view transformation is 
effected at step 23. The local transformation and the 
view transrormation are mathematically similar and, 30 
when performed by matrix multiplication, the matric- 
es defining the transformations are concatanated 
into a common matrix. Thus, the local transformation 
and the viewing transformation are performed as one 
transformation from object space to viewing space. 35 
The viewing space represents a particular view of the 
total displayable space. 

Before the viewing transformation can be effect- 
ed, it is necessary to perform lighting calculations. In 
the present embodiment, this is effected by calculat- 40 
ing the inverse to the local transformation, effecting 
this inverse transformation upon the position of light 
sources, thereby effectively transforming the light 
sources into object space. Thereafter, lighting calcu- 
lations are performed in object space and the object 45 
is transformed directly from object space to viewing 
space, using the concatanated matrix. Thus, the con- 
catanated matrix used to transform the bounding box 
during the object cull, at step 45, is used again to 
transform the actual object polygons from object 50 
space into viewing space, at step 48. 

Step 49 involves the clipping of polygons which 
do not lie within the viewing space. As previously stat- 
ed, additional information is generated at step 45, 
during the object culling process, which identifies 55 
planes bounding the viewing space through which ob- 
jects intersect 

At step 50, the three-dimensional object is pro- 



jected into a two-dimensional image and at step 52, 
backface culling is performed which, in the present 
embodiment, does not require reference to the poly- 
gon normals. Step 52 provides scan conversion and 
during this process account is taken of the restricted 
size of most frame buffers which typically have only 
eight bits per pixel location. Thereafter, a two-dimen- 
sional video image is produced by raster scanning the 
frame buffer at video rate. 

Thus, it can be appreciated that the three- 
dimensional image is constructed by transforming 
object geometry into a viewing space, from which a 
particular two-dimensional view is rendered. The 
position of object geometry, lighting and view posi- 
tion is defined by matrices and, during interactive op- 
eration, it will be the values within these matrices 
which undergo modification. 

In the preferred embodiment, transformations 
are defined by matrices having four rows and four col- 
umns with four bytes of data allocated to each posi- 
tion within the matrix, facilitating floating point ma- 
nipulations. Each matrix therefore consists of 64 
bytes of data. 

In the preferred embodiment, each remote sta- 
tion 25, 26 and 27 has its own local representation of 
image geometry. The geometry may be the same, 
such that substantially the same image is seen at 
each station. Alternatively, geometry may differ sub- 
stantially or alternatively and perhaps more appro- 
priately, the geometry may differ only in more subtle 
ways. Thus, for example, each station may include 
geometry representing a chair and said chair may 
have the same unique identification name at each 
station. However, the chair may be different in design 
or may be a different color at each of the different sta- 
tions. 

As previously stated, interactive operation at a 
station results in modification to transformation ma- 
trices. The system shown in Figure 2 operates by 
transmitting new matrix data to all of the stations, 
such that an interactive modification in three- 
dimensional space at one station will result in a sim- 
ilar modification to the space at all of the other sta- 
tions. Thus, each station maintains its own version of 
a shared three-dimensional world and the amount of 
data required to maintain equivalent versions con- 
sists only of the matrices required to effect transfor- 
mations. Thus, even when connected by a channel 
having very limited bandwidth, such as a telephone 
line, each station is updated with interactive modifi- 
cations made at any of the other stations. 

Input commands from the keyboard 17 or the 
tracker ball 1 8, are arranged to provide information to 
the communications process 41 , identifying an oper- 
ator's desire to lock an object, unlock an object, mod- 
ify the position of an object, delete an object or create 
an object. The rendered output displayed to each cli- 
ent is dependent upon that clients own view position 
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within the fhree-dimensional space, identified within 
said space by a suitable object such as a camera. 
Other cameras are present within the space, identi- 
fying the view position of other clients within the net- 
work and each camera remains permanently locked 5 
to its respective client, so that the particular client 
cannot move the cameras of others and the client's 
own camera cannot be moved by the other clients. 

Before an object may be modified by a client, it 
is necessary for that object to be locked and the lock- w 
ing and unlocking of objects is controlled by the cen- 
tral server. Thus, a client may request access to an 
object, which may or may not be given by the server, 
dependent upon whether any other client has previ- 
ously locked that particular object. During operation, 15 
a client may lock as many objects as are available 
and when locked, access to these objects is not per- 
mitted by any of the other clients in the network. 

Referring to Figure 3, on entering step 41 , enquir- 
ies are made as to whether any commands have been 20 
generated by the client operator. Referring to Figure 
5, a question is asked at step 51 as to whether a re- 
quest has been made to lock an object. If this ques- 
tion is answered in the negative, a question is asked 
at step 52 as to whether a command has been made 25 
to unlock an object. If this question is answered in the 
negative, a further question is asked at step 53 as to 
whether the operator wishes to modify an object. 
Again, if this question is answered in the negative, a 
further quest ion is asked at step 54 as to whether the 30 
operator wishes to delete an object and if this ques- 
tion is answered in the negative, a question is asked 
at step 55 enquiring as to whether the operator wish- 
es to create an object If this final question at step 55 
is answered in the negative, all possible meaningful 35 
input commands have been tested, thereby complet- 
ing the communications iteration and passing control 
to step 42 in Figure 3, identified in Figure 5 by the ar- 
row returning control to the top of the loop. 

If the question asked at step 51 is answered in 40 
the affirmative, a question is asked at step 56 as to 
whether the object has already been locked. When an 
object has been locked by a client, this information is 
supplied to all other clients in the network, therefore 
a list of locked objects is retained by all clients, allow- 45 
ing the question asked at step 56 to be answered. If 
the question asked at step 56 is answered in the af- 
firmative, to the effect that the object has been 
locked, the object is not available to the client and the 
procedure terminates. 50 

If the object has not been locked and the question 
asked at step 56 is answered in the negative, a lock 
request is made at step 57 over the communications 
channel. In response to this lock request, the central 
server issues a lock message, thus a question is 55 
asked at step 58 as to whether the object has been 
locked and this question is only answered in the af- 
firmative when the lock message from the server has 



been received. The receipt of this message allows the 
processor to leave the loop surrounding step 58, al- 
lowing a further iteration to be made. 

It should be noted that, during these communica- 
tion procedures, no modifications will have been 
made to the graphics display, therefore the question 
asked at step 42 will be answered in the negative and 
control will be returned to the process shown in Fig- 
ure 5. Thus, after requesting the locking of an object, 
a further instruction may be made to modify the pos- 
ition of the object. Such an instruction results in the 
question asked at step 53 being answered in the af- 
firmative and a further question being asked at step 
59 as to whether the object to be modified has been 
locked by the client wishing to modify it. If the ques- 
tion asked at step 59 is answered in the negative, 
modification is not permitted and control is returned. 
However, if the client wishing to modify an object has 
successfully locked use of that object, data relating to 
the modifications are supplied to the central server 
at step 60. 

After making modifications, the client would wish 
to unlock the object so that other clients may gain ac- 
cess to it Again, an input command is made to this ef- 
fect which, on the next iteration, is identified by the 
question asked at step 52 being answered in the af- 
firmative. This results in a question being asked at 
step 61 as to whether the object was previously 
locked by this client. Again, if this question is an- 
swered in the negative, control is returned but if this 
question is answered in the affirmative, an instruc- 
tion is sent to the server to the effect that the object 
is to be unlocked. Again, this information is relayed by 
the server to all other clients within the network. 

The question asked at step 54 allows a client to 
delete an object and again the question is asked at 
step 63 as to whether the object has previously been 
locked by the client wishing to make the request to de- 
lete. If this question is answered in the affirmative, 
deletion instructions are generated at step 64, which 
are supplied to the central server, thereby deleting 
the objectf ram the space under consideration. Again, 
the server will supply this information to all other cli- 
ents. 

A similar procedure is provided in response to the 
question asked at step 55 which, when answered in 
the affirmative, allows an object to be created at step 
65. 

Operation of the serving processor 37 is detailed 
in Figure 6. In a preferred embodiment, server 37 is 
only concerned with the communication of data be- 
tween the clients and does not itself have to perform 
a rendering operation. 

Not being required to perform rendering allows 
the serving processor 37 to wait for external commu- 
nications at step 71 . Upon receiving information from 
a client at step 72, enquiries are made as to the nature 
of the request, at steps 73, 74, 75, 76 and 77. If all of 
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these questions are answered in the negative, the in- 
put data is assumed to be in error and control is re- 
turned to the waiting condition at step 71. 

A request to lock an object is identified at step 73 
and results in a question being asked at step 78 as 
to whether the object has already been locked. This 
question should be answered in the negative, given 
that a request to lock a previously locked object would 
have been rejected at step 56 in Figure 5, thereby pre- 
venting a lock request from being sent. On answering 
the question raised at step 78 in the negative, a com- 
mand is sent back to the client machine at step 79, 
confirming the lock and control is returned to the wait 
condition at step 71. 

A request to modify an object is identified at step 
75 and results in a question being asked at step 80 
as to whether the object was previously locked by the 
requesting client. Again, this question would normally 
be answered in the affirmative, given that a similar 
check is made at step 59. On the question at step 80 
being answered in the affirmative, communication is 
permitted at step 81 , allowing the modifications to be 
effected. 

Requests to u n lock a n object are ident i fed at step 
74 and again a check is made at step 82 as to whether 
the object was previously locked by the client re- 
questing an unlock. Again this question should be an- 
swered in the affirmative, given that a similar check 
is raised at step 61 and thereafter an unlock com- 
mand is generated at step 83. 

Deletion requests are identified at step 76 and 
again a question is asked at step 84 as to whether the 
client requesting deletion has previously locked use 
of the object. On the question raised at step 84 being 
answered in the affirmative, communication relating 
to the deletion is permitted at step 85. 

Object creation request are identified at step 77 
and communication relating to the creation of objects 
is carried out at step 86. 

Typically, modems 34, 35, 36, 38, 39, 40 provide 
transmission at 1600 bytes per second, which allows 
one selected object to be updated twenty times per 
second. Time stamping information is added to each 
update, thereby ensuring that updates are re-se- 
quenced, if necessary at a receiver into the correct 
order. Thus, the correct interval between each up- 
date can be re-established at the receiving station in- 
dependently of the transmission speed, thereby 
faithfully reproducing an original movement. 

It is important that communications requesting 
access to an object and communications relinquish 
the locking of an object are conveyed accurately us- 
ing reliable codes, such as TCP-IP. However, given 
that a complete matrix defining a complete transfor- 
mation is transmitted, it is allowable for data relating 
to matrices to be lost, given that the correct data will 
be sent on the next iteration. Furthermore, it is pos- 
sible for terminals and communication channels to 



operate at different speeds, wherein some terminals 
will receive all iterations while others recieve a lower 
number of samples. 

5 

Claims 

1. Apparatus for displaying three-dimensional im- 
ages, comprising 

10 a first station (25) having a processing de- 

vice (15) and a display device (19) and 

a second station (26) having a processing 
device and a display device, wherein 

modifications made within a three- 

15 dimensional space displayable by said first sta- 
tion result in similar modifications to similar 
space displayable by said second station, char- 
acterised in that 

each station including means for generat- 

20 ing a video image from three-dimensional data 
and, on effecting a modification within the space 
displayable by said first station, means for trans- 
mitting modified three-dimensional data to said 
second station over a communication channel. 

25 

2. Apparatus according to claim 1, wherein said 
three-dimensional data includes geometry data 
defining objects and transformation data defin- 
ing the position of said objects, wherein said 

30 transformation related data is transmitted. 

3. Apparatus for displaying three-dimensional im- 
ages according to claim 2, wherein complete new 
transformation data is transmitted for a selected 

35 object or light source. 

4. Apparatus according to claim 2, wherein said 
transformation data represents matrices. 

40 5. Apparatus according to claim 1, wherein said 
communication channel forms part of a tele- 
phone network (PSTN). 

6. Apparatus according to claim 1, wherein a plur- 
45 ality of remote stations (25, 26, 27) are connected 
by a central server (37), wherein said server con- 
trols the access permitted by said remoted 
users. 

50 7. A method of displaying three-dimensional im- 
ages, wherein modifications within a three- 
dimensional space displayable by a first station 
result in similar modifications to similar space 
displayable by a second station, each station be- 

55 ing arranged to generate a video image from 
three-dimensional data and, on effecting a mod- 
ification within the displayable space, to transmit 
modified three-dimensional data to said second 
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* statiorf over a communication channel. 

8. A method according to claim 7, wherein said 
three-dimensional data includes geometry data 
defining objects and transformation data def in- 5 
ing the position of objects, wherein said transfor- 
mation related data is transmitted. 

9. Amethod according to claim 8, wherein complete 

new transformation data is transmitted for a se- 10 
lected object or light source. 

10. A method according to claim 8, wherein said 
transformation data represents matrices. 

15 

11. A method according to any of claims 7 to 10, 
wherein the communication channel forms part 
of a telephone network. 

12. A method according to any of claims 7 to 11, 20 
wherein a plurality of remote stations are con- 
nected by a central server and said server con- 
trols the access permitted by remote users. 

25 
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FIG. 6 OPERATION OF SERVER MACHINE 
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